Secure messaging
An optional trusted channel called the Secure Messaging System (SMS) can be enabled. It is disabled by default. This system enables applications to securely communicate with HSMs over the PCI bus interface, or across a network.
A trusted channel is created on-demand by the operator but can be terminated by either the HSM or the operator. Either the HSM or application can be configured to require a trusted channel to be created before cryptographically sensitive services can be provided. For the HSM to be FIPS-compliant, it must be configured in this way. However it is also possible for the application to request and use a trusted channel even though the HSM is not configured to require them.
The HSM can manage multiple simultaneous trusted channels, each of which will have its own set of derived session keys for message encryption/decryption and message signing/verification.
ProtectServer 3 HSMs exchange ProtectServer Identity Certificates to create trust relationships. The PTK client trusts the HSM by storing its PIC in the client's trust store.
To configure and enable SMS
-
Ensure that the PTK client has established trust with the ProtectServer 3 HSM.
You must create a ProtectServer Identity Certificate on the HSM and export it to the trust store of any PTK client that will communicate with it via Secure Messaging. See ProtectServer owner and identity certificates.
-
Configure session protection and enable SMS.
The SMS is enabled by setting one or more security flags that control how the SMS functions. By default these flags are cleared so SMS is disabled. To enable and configure SMS, see the section Configuring session protection.
-
Export the HSM Identity Certificate Chain to the appliance.
If you have enabled SMS on a ProtectServer 3 External or ProtectServer 3+ External, you must also export the certificate chain from the internal HSM module to the appliance trust store, or certain PSESH commands will not work properly. See Export the certificate chain to the ProtectServer appliance.
Configuring session protection
When applications establish a session with an HSM using ProtectToolkit-C, secure messaging layer activation depends upon:
-
Security flag settings (the security policy) stored in tamperable memory inside the HSM by the administrator
-
Any additional security flag settings specified by users where they wish to increase the level of security used. These user specified security flag settings are stored in the Secure Messaging Policy Register (SMPR) on the client machine.
Generally, the HSM-stored security flag settings are sufficient so the Secure Messaging Policy Register is rarely used.
Note
Session protection is only applied to Cryptoki functions that use a session handle returned from a previous call to C_OpenSession().
HSM-stored security flags
HSM stored security flags can be set at the local machine regardless of whether the HSM is located in the same machine as the application (PCI mode) or remotely (network mode). In the latter case it will be necessary to know the administrator’s password for the server machine as this must be entered before any server side changes can be made.
The following table lists those flags that, when set for HSM storage, affect secure messaging. For further information about these flags please see Security policies and user roles.
| Flag | Secure Messaging Effect |
|---|---|
| No clear PINs | Only messages sent to the HSM that contain sensitive data are encrypted |
| Auth Protection | Only messages sent to the HSM are signed |
| Full Secure Message Encryption | All messages sent to and from the HSM are encrypted |
| Full Secure Message Signing | All messages sent to and from the HSM are signed |
To set HSM-stored security flags
These flags can be set using ctconf -f<flags>. Refer to Security policies and user roles for full details on security policies, setting flags and the use of this command.
SMPR security flags
The Secure Messaging Policy Register (SMPR) flag settings augment the HSM settings discussed above and are stored on the client machine by assigning configuration item values.
As the client can access more than one HSM the SMPR can store a unique set of settings for each accessible HSM if required. Each HSM is identified by its serial number for SMPR storage purposes.
To set SMPR security flags
-
Obtain the serial number of the HSM.
This can be done by running ctconf -a<device> from a command line, where <device> is the number of the HSM in the list of HSMs.
-
Create the following configuration item:
...where <serial> is the serial number of the HSM found in step 1.
This change can be made at the temporary, user or system levels on both Unix and Windows platforms. Refer to Configuration items for further details on how to go about this if required.
-
Set one or more flags by assigning a value to the configuration item. For more information about the valid values for this configuration item, refer to Secure messaging configuration items. For example, if both Auth Protection and Auth Replies are required, assign the value SR.
Export the certificate chain to the ProtectServer appliance
If you have enabled SMS on a ProtectServer 3 External or ProtectServer 3+ External, you must also export the HSM Identity Certificate chain to the appliance trust store, so that the HSM can communicate securely with the appliance housing it. This procedure must be completed by each active PSESH user: admin, pseoperator, and/or audit.
To export the certificate chain to the ProtectServer appliance
-
Log on to the appliance as admin, pseoperator, or audit.
-
Fetch the cert chain from the HSM to the appliance trust store for the current user.
psesh:> hsm cert
-
Repeat this procedure for any other active appliance users.
